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ABSTRACT 

The digital environment has had an immense effect on American society, learning, and education: we have more sources 
available at our fingertips than any previous generation. Teaching and learning with these new sources, however, has 
been a challenging transition. Students are confronted with an ocean of digital objects and need skills to navigate the 
World Wide Web and numerous proprietary databases. Writing and disciplinary habits of mind are more important than 
ever in this environment, so how do we teach these in the digital age? This paper examines the current digital 
environment that humanities faculty face in their teaching and explores new tools that might support collaborative writing 
and digital skills development for students. In particular, this paper considers the effectiveness of a specially configured 
multi-agent wiki system for writing in a large lecture humanities course and explores the results of its deployment over 
two years. 
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1. INTRODUCTION 

Students in entry-level courses at large public universities are waiting for a new model of collaborative 
learning. They hope to engage with the rich digital media all around them and to participate in these courses 
more actively. Students expect, but often cannot find, courses that build digital materials around their 
increasingly collaborative learning styles. We developed a demonstration curriculum — enriched and 
supported through our multi-agent technology — where students in these courses work exclusively with digital 
objects and produce domain specific reflection, collaborative writing, deep engagement with the subject, and 
ongoing self-assessment. We wanted to create a new classroom for large lecture courses where there is less 
anonymity and more dynamic learning, where students truly participate in a "community of scholars." 

The goal of our ClassroomWiki experiment was two-fold. First, we wanted to investigate the advantages 
and disadvantages of using ClassroomWiki in a large classroom for implementing collaborative writing 
activities. To be specific, we wanted to evaluate the ClassroomWiki' s capability of: (1) providing accurate 
and detailed student interaction data to the teacher - leading to student evaluations that better reflect the 
level/amount of contribution towards their groups, (2) forming student groups that yield higher student 
participation and collaborative learning outcomes. To conduct our experiment, we divided the 150 students in 
a standard U.S. History survey course into two sets: control and treatment. Both control and treatment 
students used the same ClassroomWiki interface to collaboratively write essays on a given topic. However, 
the student groups in the control set were formed randomly, and the treatment set student groups were formed 
using MHCF, a multiagent coalition formation algorithm. 

In brief, our results indicate that ClassroomWiki' s treatment set students interacted more with the 
ClassroomWiki site, earned slightly higher scores than the control set students, and rated their peers slightly 
better compared to the control set students. Furthermore, ClassroomWiki was able to track more detailed 
information regarding student activities and provided several insights into the collaboration and interaction 
patterns of the participating students. In addition, the student scores calculated by ClassroomWiki’ s detailed 
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student contribution summary (e.g., the summary of student activities) closely represented their performances 
in other tests/assignments in the class. Finally, in the responses to our survey and self-assessment, the 
students slightly favored ClassroomWiki over Blackboard's wiki, provided several constructive comments 
that will help us improve the user-friendliness and usefulness of ClassroomWiki further, and experienced 
affective change in the ways they think about the teaching and learning of history. 


2. LEARNING ABOUT DIGITAL HISTORIES 

The problem in most large lecture courses is widely recognized. Students are assigned little writing and have 
few opportunities for engagement with the disciplinary knowledge they will need to be successful. Moreover, 
"course management software" only contributes to the distance that students feel from the process of learning 
in the discipline since these systems offer one-way delivery of ever increasing amounts and kinds of content. 
There is a significant gap in these courses between content and assessment, between the habits of mind we 
hope to teach and the activities that students perform. This is particularly true in history, where large numbers 
of students in entry-level surveys are taught through lectures, textbook assignments, and multiple choice 
tests. 

Our idea was to create a learning system that spurs collaborative approaches— groups of students will be 
able to annotate or highlight new additions to digital objects, make changes, sort, track, and compare 
versions, as well as rate and evaluate all aspects of their group's work. Rather than write analog essays that 
refer to digital objects (current practice), we envisioned students' writing digital histories in a digitally native 
system. We also envisioned collaboration that was active, meaningful, and sustained through the practice of 
joint research and writing. In this virtual space historical knowledge and thinking can be formed through 
inquiry, description, analysis, discussion, argument, and collaboration. 

Our vision was in large part an extension of our longstanding research efforts in digital history and more 
broadly digital humanities. Ayers (1999) pointed out in "The pasts and futures of digital history" that "as 
rapid as the changes have been . . . the actual writing of history has remained virtually untouched and 
unchanged. New technology has not affected the books and articles that form the foundation of what we 
teach." Much of this statement remains true today — historians continue to write history in books and articles 
in the same way — but the foundation of what historians teach has shifted. History teachers are increasingly 
referring to materials online, weaving them into their assignments, and recognizing the importance of 
primary source instruction using digital sources. Students, of course, regularly turn first to online searches for 
historical information. 

Ayers (1999) went on to call historians to a new form of digital writing: "to imagine forms of narrative on 
paper that convey the complexity we see in the digital archives." Digital history, he pointed out, "could be 
both a catalyst and a tool in the creation of a more literary kind of history." Ayers was not alone in his 
suggestion that historians write digital histories for the digital age. Robert Damton, long a leader in this field 
and a past president of the American Historical Association, called for a "new age of the book" in 1999. He 
suggested that histories might "instead of using an argument to close a case, they could open up new ways of 
making sense of the evidence, ... a new consciousness of the complexities involved in constructing the past." 
More recently, Ayers (2013) has called for a "more radical online revolution," in which undergraduates 
participate in and produce "generative scholarship" ("Toward a More Radical Online Revolution," The 
Chronicle Review , February 4, 2013). 

In The Journal of American History "Interchange: The Promise of Digital History," Steven Mintz drew 
attention to the ways digital technologies have reshaped teaching: "It has broadened my imagination. I have 
embraced audio and visual sources because they are much more accessible than in the past. I have created 
interactive, inquiry- and problem-based teaching and learning activities on the Digital History Web site 
because new technologies encouraged me to rethink the very nature of history teaching and to reimagine it as 
active engagement." (Cohen et al. 2008) Scholars have also written about how undergraduate research might 
spur new pedagogies for humanities disciplines in the digital age (e.g., Crane 2012; Pannapacker 2013; 
Thomas 2007, Thomas and Ayers 2003; Thomas and Liu 2012). 

Digital histories also help us think differently about how we know what we know about the past. Digital 
historians seeking to reconstruct the social world of, say, nineteenth-century America depend on these 
records, but by importing them into a digital medium these historians attempt to interrelate and shape the 
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information in ways that might make invisible histories visible. They create models and visualizations about 
historical questions, and attempt to uncover patterns and relationships not otherwise obvious. If we are 
beginning to write new forms of history — digital histories — should we expect our students to do the same, to 
produce their analysis in these potentially exciting new formats? (Cohen and Rosezweig 2006, Seefeldt and 
Thomas 2009, Thomas 2007) 

The ideas behind the movement in Digital History have coincided with changes in our students' 
expectations and in our classrooms. Marilyn Lombardi (2007a, 2007b) has recently pointed out the 
difficulties in teaching survey courses — they are "juggernauts" with a "merciless trajectory", she explains, 
and depend on traditional modes of assessment. Research studies suggest clearly that content coverage may 
be less valuable than independent thinking and creativity as a means toward disciplinary understanding. But 
faculty, she also points out, know how difficult it is to move in this direction. And it is clear that a great deal 
of transformative faculty development will continue to be necessary to spur active learning pedagogy. 

Other recent studies have pointed to the problem of digital information overload, of underdeveloped 
digital literary skills. Palfrey and Gasser (2010) have raised awareness about the ways the digital generation 
consumes, but does not necessarily produce, digital media. Similarly, in the field of history, Martin and 
Wineburg (2008) have indicated the need to support what they have called "the novice in the archive." They 
ask, "how do we use new digital technologies not only to make sources more available, but also to cultivate 
skills that teach students to read and think about these sources in meaningful ways?" 

At the University of Nebraska, like many land grant institutions in the U.S., the survey courses in U.S., 
European, and World history comprise an especially high demand cluster. Between 2001 and 2007 over 
25,680 students took these courses, averaging just over 3,800 students per year. The U.S. History courses 
held the majority of these students; indeed over 2,500 students on average took the U.S. survey courses a 
year. These students came from every college at the University — Art, Architecture, Business, Engineering, 
Natural Resources and Agriculture, and Arts and Sciences. To explore and evaluate a new model for 
developing historical thinking and writing in the large lecture U.S. History survey course, we put together a 
collaborative team. The team included faculty in Computer Science and Engineering, English, and History, as 
well as graduate students in History and Computer Science and Engineering. 


3. CLASSROOMWIKI 

ClassroomWiki has several sets of features. First, in terms of communications, it supports threaded forums 
where students can start a discussion thread and respond to discussion threads. Second in terms of 
collaborative interactions, it supports versioning-based collaborative editing. This allows contributing 
students to traverse between versions, review the changes, and collaboratively revise their Wiki essays. It 
also has a survey instrument for peer evaluation of their group activities. To further support students, 
ClassroomWiki allows for searching to retrieve archived communication (messages and discussion threads) 
and also provides automated collaboration reminders to notify students working in a group of any changes to 
their group pages. 

ClassroomWiki has a suite of instructional support features such as a viewable summary of student 
contributions, teacher announcements, and two options for forming groups: random and intelligent. The 
intelligent group formation tool uses the Multiagent Human Coalition Formation (MHCF) framework (based 
on the principles described in (Khandaker and Soh 2007) to form heterogeneous student groups using the 
data tracked in the ClassroomWiki environment. The novelty of this MHCF framework is in its group 
formation process which, due to its design and implementation, 1.) adapts to the changing behavior of the 
students and 2.) balances the heterogeneity of the members so that a student group contains students of all 
levels of performance. That is, from one assignment to the next, MHCF is able to re-assign students into 
groups for the next assignment based on how they behaved or collaborated in the previous assignment. The 
behaviors tracked include online activities such as the number of contributions, amount of words contributed, 
quality of participation in threaded discussions, and peer evaluation scores. When assigning students to 
groups, MHCF adopts a heterogeneity principle (Khandaker and Soh 2010): The group formation algorithm 
should balance heterogeneity of learner expertise in a group in such a way that they are less likely to give 
rise to situations where the participating learners would be de-motivated due to too high or too low 
incongruity between their expertise and the Wiki artifacts they are working on. 

For further details on ClassroomWiki, readers are referred to (Khandaker et al. 2010; Khandaker and Soh 

2010 ). 
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4. WRITING ASSIGNMENTS 

In Spring 2009 and 2010 the lead History faculty member taught a U.S. survey course to approximately 150 
students and introduced group writing assignments in the wiki format — asking the groups in each assignment 
to use library accessed databases with primary source documents (Proquest Historical Newspapers) to 
research a topic, and to collaborate to write a thesis-driven essay explaining its historical significance based 
on and from their sources. We attempted to move our approach to digital history as a research enterprise 
directly into the classroom. The specially designed wiki system, called ClassroomWiki, would allow us a 
finer granularity of analysis and with it we could test the effectiveness of the group-building algorithm and of 
the step-by-step instruction in historical reading and analysis or "habits of mind." Our pedagogy of digital 
history was built around the following concepts: 1.) emphasizing writing as a process of learning and 
disciplinary thinking, 2.) using digital primary sources to expose students to the complexity and contingency 
of the past, and 3.) using group collaboration to formulate interpretive analysis of the primary sources. 

The writing assignments were carefully sequenced. We began by helping students become better readers 
of primary documents and by giving them writing prompts for their reading. These prompts were the kinds of 
questions historians ask habitually: who wrote this, who were they writing to, what did they want or expect, 
what keywords — i.e., important words or subjects in the text — were used, what themes — i.e., broad concepts 
that the author or creator is using and emphasizing throughout the text to make his/her point — were adopted, 
and what rhetorical strategies — i.e., techniques the author or creator is using to convince, persuade, distort, or 
arrange evidence in his/her favor — were embedded in the document? To help students understand primary 
sources, we asked them to evaluate every primary source in four steps: 

1. Sourcing — find out who wrote, created, or produced this document, when, why and for what audience. 

2. Contextualizing — situating the document and the events it reports in place and time. 

3. Corroborating — checking other documents and multiple sources to assess the reliability of its claims and 
perspective. 

4. Close Reading — reading carefully to consider what a source says and the language used to say it. 


5. RESULTS 

For each group writing assignment using the wikis, we built in required steps for the teamwork. Students 
were asked to evaluate primary sources using the above criteria, conduct research using digital databases to 
find primary sources related to their topic, formulate a thesis, and complete a written paper. Students worked 
with entirely digital sources and could mix and remix them into their analysis. For a given assignment hey 
conducted their initial collaborative work in ClassroomWiki's "sandbox" where they could assemble 
materials for the group to evaluate and share. They were doing history. 

Because our ClassroomWiki system had the ability to track individual level work at a high level of 
granularity, there was a correspondingly higher degree of individual accountability. Our team's main interest 
was in the viability of the group formation agents and whether the algorithm helped groups achieve better 
performance. We found that the multi-agent system did result in slightly but statistically significant 
improvement in a group's success when compared with a control group. With more iteration of group 
activities the algorithm would probably have had a greater and deeper positive effect. 

The ClassroomWiki system also provided instructors with some novel and useful ways to see the work as 
it progressed. The system's grading and commenting functions were especially useful for our assessments. 
ClassroomWiki tracked the total word counts of individual students (average was 5,596), the numbers of 
messages posted, and. especially, the number of days each student logged onto the system and made changes 
to his or her assignment (average 6.4, high 22 days, low 2). The treatment groups earned higher average 
scores and a lower standard deviation than the control group, suggesting that treatment groups were better 
able to work together than if they were randomly formed. For both the treatment groups and the control 
groups, the total interaction numbers (counts) were remarkably similar, but the treatment group members 
rated their peers (quality of participation) slightly better. 
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Table 1 shows the correlations between the students’ scores in the ClassroomWiki assignment and their 
scores in the other tests/assignments in the class. The values in Table 1 indicate that except for the first 
document analysis assignment, the scores the students received in the ClassroomWiki assignment were well 
correlated with their scores in the other assignments/exams. These high correlation values suggest that 
individual student scores that were calculated based on ClassroomWiki’ s student contribution summary (e.g., 
number of words added/deleted, number of forum messages posted, etc.) closely represented the actual 
performance of the students in the other tests and assignments in the class. Furthermore, the correlation 
values in Table 1 indicate that the students’ final exam scores were higher for their ClassroomWiki’ s score 
than their first assignment in Blackboard's Wiki scores. Since the final exam score represents the knowledge 
and understanding of the students that they gained in the class, the higher correlation suggests that the 
detailed tracking of student behavior in ClassroomWiki allowed it to capture the performance of the students 
more closely than Blackboard’s Wiki. 

Table 1. Correlation Values 


Test/Assignment 

Correlation w / 
ClassroomWiki Scores 

Correlation w / Progressive Era 
Group Wiki (Blackboard Wiki) 
Scores 

Final 05/01/09 

0.69 

0.54 

Midterm Essay Exam 03/02/09 

0.52 

0.67 

Civil Rights Essay 03/13/09 

0.51 

0.39 

Origins of Segregation Document Analysis 1/13/09 

0.30 

0.18 

Progressive Era Group Wiki (Blackboard Wiki) 02/27/09 

0.29 



The best way to verify ClassroomWiki’ s impact on the performance of the students would be to have pre- 
test and post-test before and after the students have completed ClassroomWiki’ s assignment to test their 
knowledge of the Wiki topic. However, we did not conduct such tests, and thus it is difficult to verify the 
impact of ClassroomWiki on the students’ performance/learning. As a result, we have compared the change 
in students’ performances before and after the first Blackboard Wiki assignment with the change in students’ 
performances before and after second ClassroomWiki assignment to estimate the impact of ClassroomWiki 
on students’ performance and learning. Table 2 shows the result of our comparison. 

Table 2. Change in Students’ Scores. Note that the median for BlackboardWiki’s impact was 0.00 because there was no 
significant improvement in the control set students from the viewpoint of the median value. 

Mean Median 

BlackboardWiki’s Impact: Deltal (Midterm Score - Document Analysis Score) 3.73 0.00 

ClassroomWiki’s Impact: Delta2 (Final Exam Score - Midterm Exam Score) 2.71 3.00 


When we perform a t-test on the students’ scores, we see that the average change in students’ 
performances due to Blackboard’s Wiki and ClassroomWiki were not statistically significant. However, the 
median of the change of the students’ scores improved a little after they completed the ClassroomWiki 
assignment. This slight improvement in the median of the score change of the students suggests that 
ClassroomWiki might have helped some of the low-performing students to perform at a higher level and thus 
improved the median score. Since, our results are not conclusive, we need to collect more data using pre- and 
post- tests in our next experiment to better understand the impact of ClassroomWiki on the performance and 
learning of students. 

We also categorized the students according to the change in their scores from the midterm to the final 
exam. Table 3 shows our results. 


Table 3. Student Scores in ClassroomWiki Assignment. Note that the median for the Control Set Students was 0 because 
there were quite few control set students who did not complete/submit assignments and received a 0. 



Control Set Students 

Treatment Set Students 

All Students 



Median 

Mean 

Median 

Mean 

Median 

Mean 

Students with No change in the 
scores (Final - Midterm) 

0.00 

31.17 

74.00 

59.42 

69.00 

50.00 

Students with Positive Change 
(Final - Midterm) 

85.00 

80.21 

83.00 

82.12 

85.00 

81.16 
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The results of our analysis show that the students whose performances (i.e., scores) improved from the 
midterm to the final (second row of Table 3) had achieved higher mean (difference statistically significant 
using t-test with 5%) and higher median scores in the ClassroomWiki assignment than the students 
whose performances: (1) did not change or (2) deteriorated. 

In self-assessment and self-evaluation surveys at the end of the semester, students expressed a high level 
of affective change. They realized significant gains in historical thinking, analysis, and writing. One student 
wrote, "It’s one thing to read a history text book. It’s quite another to play the part of historian and to develop 
an opinion about historical events. This class I felt was very hands on in that enabled me to be a historian for 
a little while." Another explained, "I feel that the document analysis assignments and the wiki assignments 
really helped advance my skills in writing historical analysis. I have never been in a class that required me to 
write historical analyses like this class did." Students noted that group-writing assignments were valuable 
because they expected future employers to emphasize team project and team building. "I know that my 
ability to read primary documents and analyze them has increased immensely," one student explained, "I feel 
that my writing quality improved throughout the semester as you assigned more papers regarding analyzing 
primary documents." 

But students also expressed considerable frustration with learning historical habits of mind. Students 
expected a history course to cover events, names, and dates. They expected to be assessed on their capacity to 
recall these terms and perhaps to explain their significance. As a result there was a considerable gap between 
what students expected the survey course to be and what we were asking them to do. One student expressed 
the friction between these expectations in this way: "First of all, this course was not at all what I expected. I 
expected a ‘standard’ history class where the information is presented, notes are taken, . . . While at first 1 did 
not like the format of the class, I actually grew to like it and really appreciate what this course was trying to 
teach me in terms of historical facts but also the critical skills it forced me to acquire and utilize." Another 
student noted, "In the beginning I was very intimidated by the document analysis projects and the wikis. I 
had no idea how to analyze historical documents, considering things like perspective, tone, audience etc. . . " 
We found, therefore, an important though cautious correlation. When students were divided into three 
groups — those who improved from the mid-term exam to the final exam grade, or did not, or stayed the 
same — we found that performance on the second wiki correlated with individual exam performance (using t- 
test). Significantly, the correlation held for students from all grade ranges — that is, if they improved from C- 
to C+, B to A, or D to C. All improvers, in other words, from the individually assessed mid term exam to the 
individually assessed final exam were parts of higher scoring Wiki groups whose make-up had been tailored 
by the ClassroomWiki algorithm, and the relationship was not accidental. 

The distribution and size of the bars in Figures 1 and 2 suggest that in general, the students were able to 
complete their assignment in ClassroomWiki. Also, a higher number of students in the control group did not 
contribute to their respective groups and received a 0. This could be an indication of our group formation 
algorithm improving student groups’ collaboration by bring students of varying competence together so that 
no group is formed with students who are unable to collaborate (due to lack of competence) and complete 
their task. This claim can be further strengthened by comparing the scores of the control and treatment sets’ 
groups. The results show that the treatment set groups earned higher average scores (treatment = 74.67 vs. 
control = 70.61) and had lower standard deviation (treatment = 15.51 vs. control = 27.40, p-value <0.005). 
The higher average and lower standard deviation suggest that the treatment set student groups were able to 
work more effectively together as a group. 


6. SUMMARY 

Students in large lecture humanities courses need to gain mastery over not only content knowledge but also 
disciplinary thinking. They should in these courses learn to master a process — how to be a historian, a 
scientist, a sociologist, how to inhabit a major discipline. Mastering a process requires skill development and 
iteration, and mastery once learned can be transferred and extended. We teach introductory survey courses 
not to make students know a set of facts, but to teach students the habits of mind and critical inquiry of our 
discipline. ClassroomWiki's group writing algorithm supported the development of disciplinary thinking in a 
large lecture history survey course, improved tracking of student work in groups, and may have led to 
improved performance on individual graded examinations assessing historical thinking. 
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Figure 1 . Evaluation score distribution of ClassroomWiki students 
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Figure 1. Wiki evaluation score of students in ClassroomWiki 

Future work includes conducting more studies with groups formed using different algorithms, with 
writing assignments that are more structured and spread out over an entire semester, and with additional 
features such as tagging and rating. We are also interested in how posting previous years’ Wiki artifacts 
online as examples impact student learning as well as their performance over time. 
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